Finding your "Inner-Annotator": An Experiment in Annotator Independence for Rating Discourse Coherence Quality in Essays
نویسندگان
چکیده
An experimental annotation method is described, showing promise for a subjective labeling task – discourse coherence quality of essays. Annotators developed personal protocols, reducing front-end resources: protocol development and annotator training. Substantial inter-annotator agreement was achieved for a 4-point scale. Correlational analyses revealed how unique linguistic phenomena were considered in annotation. Systems trained with the annotator data demonstrated utility of the data.
منابع مشابه
Holistic Annotation of Discourse Coherence Quality in Noisy Essay Writing
In this paper, we describe a holistic annotation scheme for coherence quality that requires little expertise on the part of the human annotator; we also present computational systems for evaluating coherence quality in essays. A formidable challenge to reliability when annotating discourse coherence comes from differences among annotators in the inferences that they draw when reading an essay. ...
متن کاملAnnotating Discourse Relations with the PDTB Annotator
The PDTB Annotator is a tool for annotating and adjudicating discourse relations based on the annotation framework of the Penn Discourse TreeBank (PDTB). This demo describes the benefits of using the PDTB Annotator, gives an overview of the PDTB Framework and discusses the tool’s features, setup requirements and how it can also be used for adjudication.
متن کاملAssessment of Different Workflow Strategies for Annotating Discourse Relations: A Case Study with HDRB
In this paper we present our experiments with different annotation workflows for annotating discourse relations in the Hindi Discourse Relation Bank(HDRB). In view of the growing interest in the development of discourse data-banks based on the PDTB framework and the complexity associated with the discourse annotation, it is important to study and analyze approaches and practices followed in the...
متن کاملApplying Syntactic, Semantic and Discourse Constraints in Chinese Temporal Annotation
We describe a Chinese temporal annotation experiment that produced a sizable data set for the TempEval-2 evaluation campaign. We show that while we have achieved high inter-annotator agreement for simpler tasks such as identification of events and time expressions, temporal relation annotation proves to be much more challenging. We show that in order to improve the inter-annotator agreement it ...
متن کاملLexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays
This paper presents an investigation of lexical chaining (Morris and Hirst, 1991) for measuring discourse coherence quality in test-taker essays. We hypothesize that attributes of lexical chains, as well as interactions between lexical chains and explicit discourse elements, can be harnessed for representing coherence. Our experiments reveal that performance achieved by our new lexical chain fe...
متن کامل